Gesture input with multiple views, displays and physics

ABSTRACT

Gesture input with multiple displays, views, and physics is described. In one example, a method includes generating a three dimensional space having a plurality of objects in different positions relative to a user and a virtual object to be manipulated by the user, presenting, on a display, a displayed area having at least a portion of the plurality of different objects, detecting an air gesture of the user against the virtual object, the virtual object being outside the displayed area, generating a trajectory of the virtual object in the three-dimensional space based on the air gesture, the trajectory including interactions with objects of the plurality of objects in the three-dimensional space, and presenting a portion of the generated trajectory on the displayed area.

CLAIM OF PRIORITY

This patent arises from a continuation of U.S. patent application Ser.No. 14/954,861, now U.S. Pat. No. 10,963,062, entitled GESTURE INPUTWITH MULTIPLE VIEWS, DISPLAYS AND PHYSICS filed Nov. 30, 2015. U.S.patent application Ser. No. 14/954,861 is a continuation of U.S. patentapplication Ser. No. 13/977,654, now U.S. Pat. No. 9,557,819, entitledGESTURE INPUT WITH MULTIPLE VIEWS, DISPLAYS AND PHYSICS, filed Jun. 28,2013. U.S. patent application Ser. No. 13/977,654 is a U.S. NationalPhase Application under 35 U.S.C. § 371 of International PatentApplication No. PCT/US2011/062140, entitled GESTURE INPUT WITH MULTIPLEVIEWS, DISPLAYS AND PHYSICS, filed Nov. 23, 2011. The benefit of andpriority to U.S. patent application Ser. No. 14/954,861, U.S. patentapplication Ser. No. 13/977,654 and International Patent Application No.PCT/US2011/062140 are hereby claimed. U.S. patent application Ser. No.14/954,861, U.S. patent application Ser. No. 13/977,654 andInternational Patent Application No. PCT/US2011/062140 are incorporatedherein by reference in their entireties.

FIELD

The present description pertains to user input methodology and displayin a computer system and, in particular, to representing user gesturesin multiple display or three-dimensional display system.

BACKGROUND

Computer system inputs have developed to include air gestures and touchscreen gestures as inputs. Air gestures can involve users moving theirbody and having a corresponding action happen on a display or having acorresponding command be executed by the computing system. One form ofcurrent air gesture technology uses movable sensors either as acontroller or as the game console. The sensors are held in the hand,attached to the body, or manipulated by hands, feet, or other parts ofthe body (such as in the Wii remote controller by Nintendo Company,PlayStation Move by Sony Corporation, and various smartphones, andhandheld gaming devices). Another form of air gesture technology uses 3Dcamera and microphone technology (such as in the Microsoft Kinect byMicrosoft Corporation and PlayStation Eye by Sony Corporation) toapproximate body motion as a modal input source.

Television, computer, and portable device displays are typical feedbackmechanisms for viewing the effects of air-gesture modal input on agraphical environment. Cameras gather video input for gesture detection,and the video input is interpreted with software that runs on a gamingconsole or a personal computer. A camera array allows the cameras tosense depth. This provides the ability to recognize the position ordistance of a person's body is in relation to the camera. The cameraarray allows for additional air gestures that move towards and away fromthe camera.

As another form of gestures, the screens of gaming consoles andcomputers, such as desktop, notebooks, tablets and smartphones,incorporate touch-screen technology which responds to touch input. Touchand sweep gestures on a display screen are used as a user input toexecute commands to move objects from one screen (e.g. a handheldconsole screen) to another screen (e.g. a TV screen). Such features areimplemented, for example, when using the PlayStation Portable gameconsole together with a Play Station 3 console both marketed by the SonyCorporation. Touch and sweep gestures are also offered on trackpads ofnotebook computers and as surfaces on a peripheral mouse or externaltrack pad.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example, and notby way of limitation, in the figures of the accompanying drawings inwhich like reference numerals refer to similar elements.

FIG. 1A is diagram of a display showing a first view of an applicationand a user gesture applied to that view according to an embodiment ofthe invention.

FIG. 1B is a diagram of a display showing a second view of theapplication and a user gesture applied to that view according to anembodiment of the invention.

FIG. 1C is a diagram of two displays simultaneously showing the firstand the second views of the application and a user gesture applied toone of the views according to an embodiment of the invention.

FIG. 2A is a diagram of a display showing a third view of theapplication and a user gesture applied to that view according to anembodiment of the invention.

FIG. 2B is a diagram of a two displays simultaneously showing differentparts of the third view of the application and a user gesture applied tothat view according to an embodiment of the invention.

FIG. 3 is a process flow diagram of interpreting a gesture based on aselected view of an application according to an embodiment of theinvention.

FIG. 4 is a process flow diagram of interpreting a gesture based on aselected view and a selected display of an application according to anembodiment of the invention.

FIG. 5 is a process flow diagram of interpreting a gesture based on aselected view and a selected display of an application according toanother embodiment of the invention.

FIG. 6 is a process flow diagram of interpreting a gesture on multipledisplays using interacting physics of an application according to anembodiment of the invention.

FIG. 7 is a process flow diagram of interpreting a gesture on multipledisplays using interacting physics of an application according toanother embodiment of the invention.

FIG. 8 is block diagram of a computer system suitable for implementingprocesses of the present disclosure according to an embodiment of theinvention.

FIG. 9 is a block diagram of a an alternative view of the computersystem of FIG. 8 suitable for implementing processes of the presentdisclosure according to an embodiment of the invention.

DETAILED DESCRIPTION

While gestures, whether air gestures or touch gestures find increasingapplication in computing environments, they lack a common feature ofpointing devices. They do not necessarily indicate where the gesture isdirected. For multiple windows, screens, or displays, it is not alwaysclear where a gesture is directed. In examples described below, acomputing system interprets a user gesture in different ways dependingon the current view that is presented by an application or a computingsystem. The computing system determines a device, window, or screen auser is facing or looking at to determine the objects at which a gestureis directed. Multiple displays can be used to show different views ofthe same application or a game simultaneously, allowing users tocoordinate gesture inputs from different perspectives. A similarapproach can be applied to voice commands.

While eye tracking can be used to pair a voice command with an object ona computer display, multiple devices may have displays that presentdifferent objects simultaneously. The displays may also present the sameobjects in a different way. The system can react to user air, touch, orvoice gestures differently depending on the current view of theapplication being displayed on a screen and depending on which screen auser is looking at. Air and voice gestures can then be directed at theappropriate view.

In addition, air, touch, and voice gestures can be used to createinteraction between virtual objects that are not displayed and elementson the screen that cause physics effects on the virtual objects. In sucha case, virtual objects can interact in a three-dimensional space infront of and behind a displayed screen plane. The displayed objects canbe displayed on any one of several different screens.

The three-dimensional space can be characterized by targets,obstructions, and fields in, for example, a computer gaming environmentin which, due to the physics characteristics of those objects, theyinteract with user gestures that are applied to virtual objects.Three-dimensional physics effects can be represented in thisthree-dimensional space. In this three-dimensional space, games andother applications can combine forces from targets, obstructions, andfields with forces from air gestures to provide a more complex,interactive, or realistic interaction with a user.

FIG. 1A is a diagram of an air gesture system having a display 101coupled to an array of cameras 103 and an array of microphones 105. Inthe illustrated example, there are two cameras and two microphones,however, a larger or smaller number of cameras or microphones may beused for more or less accurate sensing of position and direction. Thedisplay may be a direct view or projection display on any type ofdisplay technology. As shown the camera microphone arrays are positionover and attached to the display. However, any other position may beused. The camera and microphone may be positioned apart from each otherand apart from the display. The arrays can be calibrated for orconfigured with knowledge of the position of the display in order tocompensate for offset positions. The display may be a part of a portablecomputer, a gaming console, a handheld smartphone, personal digitalassistant, or media player. Alternatively, the display may be a largeflat panel television display or computer monitor

In the example shown, the display shows three submarines 109 from a sideview progressing through an undersea environment. A user shown as a hand107 performs air gestures to direct torpedoes 111 at the displayedsubmarines. The user air gestures are detected by the cameras to executea command to fire torpedoes. The system uses a gesture library for theundersea environment that contains possible gestures. When the handperforms a gesture, the system compares the observed gesture to thegesture library, finds the closest gesture, then looks up the associatedcommand, such as fire torpedoes.

FIG. 1B shows the same display 101 with the same camera and microphonearrays and the same submarines 109. However, in FIG. 1B the submarinesare viewed from the top for example from the surface of the water orfrom a shallow depth downwards towards the submarines. The user 107 isperforming the same air gesture which instead results in the release ofdepth charges 113 down toward the submarines. As can be seen, dependingon whether the view of the submarines is from the side as in FIG. 1A orfrom the top as in FIG. 1B, the same finger pinch-release gesture asillustrated can result in different actions. In the example of FIG. 1A,the user gesturing from the side can make a throw gesture with a pinchand release to cause a torpedo to go towards the targets. In FIG. 1B thesame pinch release can cause depth charges to be dropped toward thetargets on the screen. While the gestures are the same, the system candetermine the current view whether from the side or the top to determinewhether the gesture is interpreted as a release of torpedoes or as arelease of depth charges. As a result, a user can use an intuitivegesture which is simple to perform to cause different commands to beexecuted by the system.

FIG. 1C shows the same two displays side-by-side. In the illustratedexample, both displays have a camera and microphone array, however, asingle camera and microphone array may be used. These arrays may beconnected to either display or located in a different position. In thisexample, each display 101 a and 101 b show the same three submarines,one shows the submarines 109 a from the side while the other shows thesubmarines 109 b from the top. The user can either throw torpedoes ordrop depth charges 113 on the same submarines depending upon whichscreen is being used, or is current or active, at the time. As shown,the environment presents two displays that present the same threesubmarines simultaneously. The gesture, such as a pinch-release gesture,does not indicate which display the user intends, so that the systemdoes not know whether to produce the torpedo command or the depth chargecommand. In this example, the camera array on one or both of the screenscan determine which screen the user intends. For example, by trackingthe users face, eye focus, or voice direction, the system can determinewhich screen the user is focusing attention on and then activate thecorresponding command for that screen.

The same approach can also be used with touch screen and touch surfacegestures, as well as with voice commands, rather than free hand airgestures The user might have a touch screen or touch surface and performgestures on those surfaces. Again, in order to determine which view thegesture is to be applied against, the system can determine where theuser's focus is. If the user is focusing on the side view, then agesture on the touch surface can result in torpedoes being launched.Whereas if the user is focusing on the top view, the gesture can resultin depth charges being launched. The two views of FIG. 1A and FIG. 1Bcan represent two different views of a single application. In FIG. 1C,the application generates both views simultaneously, whereas in FIGS. 1Aand 1B only one view can be seen at a time. In either example, thesystem can determine the current view being used by the user and thecurrent display.

If there is only one display, then a single view is used, however, thesingle display may present different windows on the one display. Forexample, the display of FIG. 1A may be presented in one window of adisplay whereas the display of FIG. 1B may be presented in anther windowof the display. In such an example, the camera array of 103 candetermine which of the two windows the user has focused on and thenexecute the appropriate command for the user's gesture.

FIG. 2A shows a different screen display In FIG. 2A, the same display101 with the same camera 103 and microphone 105 array presents adifferent view. In this example, the user 107 is using air gestures tothrow a virtual spaceship at the screen. The spaceship appears on thescreen after it is has traveled some distance from the user's airgesture and its behavior is governed by the user's gesture such as athrow and by the objects on the screen. In the illustrated example thereis a large planet 121 surrounded by several moons 123. The user has beenpresented with a target 125 on the planet which is to be approached bythe flying spaceships 127.

In the example of FIG. 2A, each of the planets and moons have a sizerelative to each other which determines its gravitational force as thespaceships are thrown toward the target 125. The moons and the planetmodify the velocity and direction of the spaceships' travel due to theirgravitational fields. As result, the user may intend to throw thesespace ships directly at the target but they may be pulled off course bya nearby moon or they may fall into orbit around the planet or a moonand not actually reach the target directly.

In the example of FIG. 2A, the screen display may present a part of athree dimensional space that has been generated by the system. In thisthree dimensional space, objects appear in the foreground and in thebackground on the screen. These can be rendered on a three dimensionaldisplay using shutter glasses or lenticular pixels that cause theperception of objects being closer and farther from the user. Theobjects can also be rendered on a two dimensional display usingperspective. In both examples, the screen surface represents aparticular plane in a z-axis running toward or away from the user. Thescreen lies at one point on this z-axis and objects cast by the userbegin at some distance from the plane of the screen.

When a user throws an object towards the screen, it is first a virtualobject which the user cannot see. As it reaches the plane of the screenin the three dimensional space, it appears as a displayed object on thescreen. After it reaches the plane of the screen it continues to abackground that may be represented as far away points on the screen.

The interaction with objects on the screen maybe enhanced further byincluding additional objects in the three dimensional space that are notshown on the screen. As a result, a user may throw a spaceship 127toward the target 125 and find that its course and velocity have alreadybeen altered before it reaches the plane of the screen. These objectsand the alteration in course will not be shown on the screen. However,the effects will be shown when the spaceship arrives at the plane of thescreen.

FIG. 2B is a diagram of the same display and screen of FIG. 2A. However,an additional screen 131 has been added. This screen is shown as aportable device such as a smart phone or portable gaming system howeverit could be any other type of display including a display of the sametype as the main display 101. The small display 131 is placed, in thisexample, in front of the main large display 101. The system candetermine the position of the small screen 131 and present a portion ofthe three dimensional space that lies in the plane of the small screen.So for example in FIG. 2B, the user 107 has thrown a space ship 127toward the planet 121 and in particular at the target 125 on thatplanet. After the spaceship has been thrown it first appears on thesmall screen 131.

As shown an object 129 that is not visible on the main screen 101 isvisible on the small screen. This object 129 is in the form of anothermoon which can exert a gravitational or other force on the spaceship127. As the spaceship continues through the three dimensional space, itwill leave the small display 131 and after some time show up on thelarge display 101. The addition of the small screen adds a new dimensionto this particular type of game play. The camera array 103 or some otherproximity sensing system can determine the position of the small screenin real time. The user can then move the small screen around to seeobjects that are not displayed on the main screen 101. As a result, uponthrowing a space ship 127 in the example of FIG. 2A, if the course andvelocity of the spaceship is significantly altered, the user can use asmall screen 131 to find which objects have influenced its path andcompensate accordingly. The small screen can be moved around indifferent planes on the z-axis to see what is in front of the largescreen 101. A similar approach can be used to see what is beside orbehind the large screen.

The approach discussed above with respect to FIG. 1C can also be used inthe example of FIG. 2B. In the case of, for example, a smart phone thesmall screen 131 will also be equipped with one or more cameras andmicrophones facing the user. While these are typically provided for useon video conferences and telephone voice calls, the cameras andmicrophones can instead be used to determine the position of a user, theposition of other displays, and to see and interpret gestures. Similarlythe cameras on the small screen 131 and the large screen 101 can be usedto determine where a user's attention is focused and interpret airgestures or other gestures according to the particular display used. Sofor example instead of showing a different portion of a threedimensional space, the small screen 131 may instead be used to show adifferent view as in the example of FIG. 1C.

FIG. 3 is an example process flow for using the display and userconfiguration shown in FIGS. 1 and 2 . In FIG. 3 , the process startsand a user launches an application which may be a game or any otherapplication to which gestures and multiple displays may be used. At 303,the system presents a default view of the application. This default maybe determined in a variety of different ways. At 305, the systemactivates a gesture library for the default view. In this case, currentview is the default view and so this gesture library can be loaded bydefault. The gesture library can be formed in a variety of differentway. In one example, the gesture library is in the form of a look uptable in which particular camera sensor points are connected todifferent commands that the program can execute. In the examplesdiscussed above, a similar gesture can be executed as a command to throwtorpedoes, launch depth charges, or throw spaceships depending on theparticular view presented to the user. A wide range of differentgestures maybe be used in the library to perform different commands.

At 307, the system waits to determine if a gesture had been received.This gesture can be received through cameras, through a touch surface,through a touch screen, or a voice gesture can be received in amicrophone. If a gesture is received, then the process proceeds to block311, where the gesture is matched to the loaded current library. Thesystem matches the gesture to one of the gestures in the library andthen finds the corresponding command.

At 313 this command is executed and at 315 the display is modified todisplay the action on the screen of the executed command. Afterexecuting a command of the gesture, the system detects whether there hasbeen a change in view at 317. A change of view corresponds to adifferent window on the display or a different display. If a change ofview is detected, then the process proceeds to present the change inview and then returns to block 305 to change the gesture library tocorrespond to the change in the view. If no change in the view isdetected, then the system proceeds to wait for a new user gesture at307. If a user gesture is received then, as before at 311, the gestureis matched in the currently loaded library. If no gesture is receivedthen the system proceeds to 317 to determine if a change in the view isdetected. This cycle can repeat to receive additional user gestures andto view changes to provide user interaction throughout use of thesystem.

FIG. 4 shows an alternative process flow diagram for using multipleviews and multiple displays in an application. At 401, the system isstarted and an application is launched. At 403, a default view for theapplication is presented. At 405, the active display is determined. Thiscan be determined by determining a user's focus or attention. In oneexample, the camera array determines which direction a user is looking.For example, a camera can detect a face and determine an angle of theface to determine if the user is looking directly at one display or theother. This can be done, as in the example of FIG. 1C, using a separatecamera array for each display. Alternatively, a single camera array candetermine if a user is looking at one display or the other. In oneexample, the camera array determines the position of a user's pupils tosee in which direction the user is looking. In another example, thecamera array determines which way a face is pointed. Other user actionscan be used to determine which display is the active display. Forexample, the user can point to different displays, make swiping motionsin the air or a variety of other gestures to indicate which displayshould be the active display.

At 407, the gesture library for the current view and the current displayis activated. The system loads the input recognition library which isappropriate for that display and that view. At 409, the systemdetermines whether a user gesture has been received if a user gesturehas been received, then at 411 this gesture is matched to the currentlibrary. The corresponding command is executed at 413 and a modifieddisplay is generated at 415. If no user gesture has been received, thenthe process skips forward to determine if a change of view is detectedat 417. If a change of view has not been detected then the systemreturns to determine the active display at 405. If a change of view hasbeen detected, then the changed view is presented at 419 and the processreturns to determine the act of display.

The process flow of FIG. 4 allows a system to match the gesture libraryto a particular view as well as to a particular display. As a result, anapplication can present multiple views and multiple displays and changethe effect of user gestures depending on the current view and currentdisplay. In alternative embodiments, the user may be only presented withdifferent views or different displays but not both depending on theimplementation.

FIG. 5 shows a simplified process flow for using air gestures withdifferent displays. At 501, the process is started and a displayselection is received. The display selection can be done by determiningwhere a user is looking with face detection or eye tracking ordetermining in which direction a user is speaking through the microphonearray or a user can indicate a particular display through voice or aircommands. At 503, an air gesture is received. At 505, the current viewof a selected display is determined. At 507, a command is selected basedon the current view for the selected display and, at 509, the selectedcommand is executed. This process is repeated to provide repeated userinteraction with the displays and with the application in the providedviews.

Referring to FIG. 6 , applications can incorporate a variety ofinteracting physics with a gesture library to present user interactionswith different views. At 601, the application is started and anapplication is launched. At 603, a default view is presented. At 605, agesture library is activated for the current view. The relevant gesturetemplates are loaded into memory together with the correspondingcommands to be executed upon detecting the gesture.

At 607 the system determines whether any additional displays can bedetected. If so then the position of this additional display isdetermined at 621. This is calculated using cameras RF (Radio Frequency)or IR (Infrared) sensors. At 623, a view is presented on this additionaldisplay based on its position. The process returns to determine whetheruser gestures are received at 609. If no user gesture is received, then,when the process continues to wait in the background, additionalprocesses can continue to detect additional displays and to detect thecurrent view. Other processes can also run simultaneously to detectwhich display is active as described in the examples above.

When a gesture is received at 609, then, at 611, the gesture is matchedto the current library. The user can use gestures to select projectiles,launch projectiles, change configuration settings and more. When agesture has been matched against the current loaded gesture library thena command is selected and, at 613, the command is modified in accordancewith the parameters of the gesture. So, for example, the system canmeasure the velocity of the hand, the angle of movement of the hand, andthe point of release by the hand, or similar parameters using otherkinds of air gestures or touch surface gestures. These parameters arethen added to the command from the gesture library and, at 615, usingthe interacting physics, the resulting action is determined.

A virtual object launched by an air gesture is given a velocity and adirection by the user's gesture. It can also possess a virtual mass, airresistance, acceleration and other possible physics parameters. Thesystem then calculates interacting physics between the virtual objectgenerated by the gesture parameters and displayed objects in the threedimensional space. Additional interactions can be calculated for objectsthat are not displayed but which are still present in the threedimensional space. As an example, the moon 129 of FIG. 2B is an objectin the three dimensional space that is not displayed on the main screen101. A user without the additional screen 131 would not see that objectas a displayed object. However, that object can exert an interactionupon the virtual object generated by the air gesture. At 617, thesecommands are executed and, at 619, the display is modified to show thevirtual object as it arrives on the display. The display is alsomodified to show the results of its interactions with other objects inthe three dimensional space including the displayed objects and perhapsadditional objects in the three dimensional space that are notdisplayed. Having executed the command, the system returns to receiveadditional user gestures at 609.

FIG. 7 shows a simplified process flow for using a three dimensionalspace of objects and forces with user gestures. At 701, the processstarts and the application is launched. At 703, a three dimensionalspace is generated that includes one or more objects and one or moreforces. In the example of FIG. 2B, these objects are planets and moonswith gravitational forces. However a wide range of different kinds ofobjects can be generated and different kinds of forces can be used. At705, the system determines the displays that are available for use bythe system. At 707, the relative position and orientation of theseavailable displays is determined and, at 709, a portion of the threedimensional space is presented on the available displays. The amount ofthe three dimensional space that is presented on the displays can bedetermined based on the size and position of the displays as well aswhether the displays allow for the presentation of three dimensionalviews or two dimensional views. At 711, the system determines if a usergesture has been received. If not, it waits for the user gesture. If auser gesture has been received, then, at 713, a trajectory of aresulting virtual object is generated in the three dimensional space. At715, a portion of the generated trajectory is shown on the availabledisplays. As mentioned above, the launched virtual object as a result ofthe gesture may traverse part of the three dimensional space withoutbeing visible on any display and may traverse another part of the threedimensional space visible on the displays. The system can determine theposition of the virtual object as it traverses the three dimensionalspace and compare that to the portion of the three dimensional spacethat is presented on the displays that are available. In this way, theobject can enter and leave displays while still traversing a consistenttrajectory through the three dimensional space. After presenting thegenerated trajectory the process returns to receive additional usergestures at 711.

There are a wide variety of different effects and interaction that maybe provided, depending on the particular implementation. The Table showssome of these, however, embodiments of the invention are not so limited.

TABLE Virtual Projectiles Obstructions Physics (e.g., spaceships) andFields Target Gravity The projectile could Larger and Some targets causegravitational more dense could have effects on smaller (darker colored)gravitational objects such as other obstructions forces. spacecraft.will have more gravity. Acceleration The projectile could Obstructionswill Targets accelerate and cause projectiles to could deceleratedepending on accelerate toward try to elude other effects them.projectiles. Velocity Generally higher Obstructions Targets velocitywill minimize will have can be gravitational effects, the potentialmoving. though not in special to move. cases. Inertia The virtual massof the Depending on All objects projectile would interact mass,obstructions and fields with other physics could be have the elements.effected by potential projectiles or for repelling other obstructions.objects. Static Resistance fields slow Resistance fields A targetresistance progress of projectile. will generally could be fields Aerodynamics would slow projectiles within make a difference. but could astatic speed them. or initial resistance field. Elastic The projectilecould be Some Projectiles resistance launch with a virtual obstructionscould elastic band, which could have elastic bounce off would supplyadditional properties multiple physics. to bounce targets. projectiles.Inertial Some resistance fields Potential for more A target resistancecould offer more resistance to faster could fields resistance to fastermoving objects. be within moving objects. a static or initial resistancefield. Repel All objects and fields All objects All objects forces havethe potential for and fields and fields repelling objects. have the havethe potential for potential repelling objects. for repelling objects.

FIG. 8 is a block diagram of a computing environment capable ofsupporting the operations discussed above. The modules and systems canbe implemented in a variety of different hardware architectures and formfactors including that shown in FIG. 9 .

The Command Execution Module 801 includes a central processing unit tocache and execute commands and to distribute tasks among the othermodules and systems shown. It may include an instruction stack, a cachememory to store intermediate and final results, and mass memory to storeapplications and operating systems. The Command Execution Module mayalso serve as a central coordination and task allocation unit for thesystem.

The Screen Rendering Module 821 draws objects on the one or moremultiple screens for the user to see. It can be adapted to receive thedata from the Virtual Object Behavior Module 804, described below, andto render the virtual object and any other objects and forces on theappropriate screen or screens. Thus, the data from the Virtual ObjectBehavior Module would determine the position and dynamics of the virtualobject and associated gestures, forces and objects, for example, and theScreen Rendering Module would depict the virtual object and associatedobjects and environment on a screen, accordingly. The Screen RenderingModule could further be adapted to receive data from the Adjacent ScreenPerspective Module 807, described below, to either depict a targetlanding area for the virtual object if the virtual object could be movedto the display of the device with which the Adjacent Screen PerspectiveModule is associated. Thus, for example, if the virtual object is beingmoved from a main screen to an auxiliary screen, the Adjacent ScreenPerspective Module 2 could send data to the Screen Rendering Module tosuggest, for example in shadow form, one or more target landing areasfor the virtual object on that track to a user's hand movements or eyemovements.

The Object and Gesture Recognition System 822 may be adapted torecognize and track hand and harm gestures of a user. Such a module maybe used to recognize hands, fingers, finger gestures, hand movements anda location of hands relative to displays. For example, the Object andGesture Recognition Module could for example determine that a user madea body part gesture to drop or throw a virtual object onto one or theother of the multiple screens, or that the user made a body part gestureto move the virtual object to a bezel of one or the other of themultiple screens. The Object and Gesture Recognition System may becoupled to a camera or camera array, a microphone or microphone array, atouch screen or touch surface, or a pointing device, or some combinationof these items, to detect gestures and commands from the user.

The touch screen or touch surface of the Object and Gesture RecognitionSystem may include a touch screen sensor. Data from the sensor may befed to hardware, software, firmware or a combination of the same to mapthe touch gesture of a user's hand on the screen or surface to acorresponding dynamic behavior of a virtual object. The sensor data maybe used to momentum and inertia factors to allow a variety of momentumbehavior for a virtual object based on input from the user's hand, suchas a swipe rate of a user's finger relative to the screen. Pinchinggestures may be interpreted as a command to lift a virtual object fromthe display screen, or to begin generating a virtual binding associatedwith the virtual object or to zoom in or out on a display. Similarcommands may be generated by the Object and Gesture Recognition Systemusing one or more cameras without benefit of a touch surface.

The Direction of Attention Module 823 may be equipped with cameras orother sensors to track the position or orientation of a user's face orhands. When a gesture or voice command is issued, the system candetermine the appropriate screen for the gesture. In one example, acamera is mounted near each display to detect whether the user is facingthat display. If so, then the direction of attention module informationis provided to the Object and Gesture Recognition Module 822 to ensurethat the gestures or commands are associated with the appropriatelibrary for the active display. Similarly, if the user is looking awayfrom all of the screens, then commands can be ignored.

The Device Proximity Detection Module 825 can use proximity sensors,compasses, GPS (global positioning system) receivers, personal areanetwork radios, and other types of sensors, together with triangulationand other techniques to determine the proximity of other devices. Once anearby device is detected, it can be registered to the system and itstype can be determined as an input device or a display device or both.For an input device, received data may then be applied to the ObjectGesture and Recognition System 822. For a display device, it may beconsidered by the Adjacent Screen Perspective Module 807.

The Virtual Object Behavior Module 804 is adapted to receive input fromthe Object Velocity and Direction Module, and to apply such input to avirtual object being shown in the display. Thus, for example, the Objectand Gesture Recognition System would interpret a user gesture and bymapping the captured movements of a user's hand to recognized movements,the Virtual Object Tracker Module would associate the virtual object'sposition and movements to the movements as recognized by Object andGesture Recognition System, the Object and Velocity and Direction Modulewould capture the dynamics of the virtual object's movements, and theVirtual Object Behavior Module would receive the input from the Objectand Velocity and Direction Module to generate data that would direct themovements of the virtual object to correspond to the input from theObject and Velocity and Direction Module.

The Virtual Object Tracker Module 806 on the other hand may be adaptedto track where a virtual object should be located in three dimensionalspace in a vicinity of an display, and which body part of the user isholding the virtual object, based on input from the Object and GestureRecognition Module. The Virtual Object Tracker Module 806 may forexample track a virtual object as it moves across and between screensand track which body part of the user is holding that virtual object.Tracking the body part that is holding the virtual object allows acontinuous awareness of the body part's air movements, and thus aneventual awareness as to whether the virtual object has been releasedonto one or more screens.

The Gesture to View and Screen Synchronization Module 808, receives theselection of the view and screen or both from the Direction of AttentionModule 823 and, in some cases, voice commands to determine which view isthe active view and which screen is the active screen. It then causesthe relevant gesture library to be loaded for the Object and GestureRecognition System 822. Various views of an application on one or morescreens can be associated with alternative gesture libraries or a set ofgesture templates for a given view. As an example in FIG. 1A apinch-release gesture launches a torpedo, but in FIG. 1B, the samegesture launches a depth charge.

The Adjacent Screen Perspective Module 807, which may include or becoupled to the Device Proximity Detection Module 825, may be adapted todetermine an angle and position of one display relative to anotherdisplay. A projected display includes, for example, an image projectedonto a wall or screen. The ability to detect a proximity of a nearbyscreen and a corresponding angle or orientation of a display projectedtherefrom may for example be accomplished with either an infraredemitter and receiver, or electromagnetic or photodetection sensingcapability. For technologies that allow projected displays with touchinput, the incoming video can be analyzed to determine the position of aprojected display and to correct for the distortion caused by displayingat an angle. An accelerometer, magnetometer, compass, or camera can beused to determine the angle at which a device is being held whileinfrared emitters and cameras could allow the orientation of the screendevice to be determined in relation to the sensors on an adjacentdevice. The Adjacent Screen Perspective Module 807 may, in this way,determine coordinates of an adjacent screen relative to its own screencoordinates. Thus, the Adjacent Screen Perspective Module may determinewhich devices are in proximity to each other, and further potentialtargets for moving one or more virtual object's across screens. TheAdjacent Screen Perspective Module may further allow the position of thescreens to be correlated to a model of three-dimensional spacerepresenting all of the existing objects and virtual objects.

The Object and Velocity and Direction Module 803 may be adapted toestimate the dynamics of a virtual object being moved, such as itstrajectory, velocity (whether linear or angular), momentum (whetherlinear or angular), etc. by receiving input from the Virtual ObjectTracker Module. The Object and Velocity and Direction Module may furtherbe adapted to estimate dynamics of a any physics forces, by for exampleestimating the acceleration, deflection, degree of stretching of avirtual binding, etc. and the dynamic behavior of a virtual object oncereleased by a user's body part. The Object and Velocity and DirectionModule may also use image motion, size and angle changes to estimate thevelocity of objects, such as the velocity of hands and fingers

The Momentum and Inertia Module 802 can use image motion, image size,and angle changes of objects in the image plane or in athree-dimensional space to estimate the velocity and direction ofobjects in the space or on a display. The Momentum and Inertia Module iscoupled to the Object and Gesture Recognition System 822 to estimate thevelocity of gestures performed by hands, fingers, and other body partsand then to apply those estimates to determine momentum and velocitiesto virtual objects that are to be affected by the gesture.

The 3-D Image Interaction and Effects Module 805 tracks user interactionwith 3-D images that appear to extend out of one or more screens. Theinfluence of objects in the z-axis (towards and away from the plane ofthe screen) can be calculated together with the relative influence ofthese objects upon each other. For example, an object thrown by a usergesture can be influenced by 3-D objects in the foreground before thevirtual object arrives at the plane of the screen. These objects maychange the direction or velocity of the projectile or destroy itentirely. The object can be rendered by the the 3-D Image Interactionand Effects Module in the foreground on one or more of the displays.

FIG. 9 is a block diagram of a computing system, such as a personalcomputer, gaming console, smartphone or portable gaming device. Thecomputer system 900 includes a bus or other communication means 901 forcommunicating information, and a processing means such as amicroprocessor 902 coupled with the bus 901 for processing information.The computer system may be augmented with a graphics processor 903specifically for rendering graphics through parallel pipelines and aphysics processor 905 for calculating physics interactions as describedabove. These processors may be incorporated into the central processor902 or provided as one or more separate processors.

The computer system 900 further includes a main memory 904, such as arandom access memory (RAM) or other dynamic data storage device, coupledto the bus 901 for storing information and instructions to be executedby the processor 902. The main memory also may be used for storingtemporary variables or other intermediate information during executionof instructions by the processor. The computer system may also include anonvolatile memory 906, such as a read only memory (ROM) or other staticdata storage device coupled to the bus for storing static informationand instructions for the processor.

A mass memory 907 such as a magnetic disk, optical disc, or solid statearray and its corresponding drive may also be coupled to the bus of thecomputer system for storing information and instructions. The computersystem can also be coupled via the bus to a display device or monitor921, such as a Liquid Crystal Display (LCD) or Organic Light EmittingDiode (OLED) array, for displaying information to a user. For example,graphical and textual indications of installation status, operationsstatus and other information may be presented to the user on the displaydevice, in addition to the various views and user interactions discussedabove.

Typically, user input devices, such as a keyboard with alphanumeric,function and other keys, may be coupled to the bus for communicatinginformation and command selections to the processor. Additional userinput devices may include a cursor control input device such as a mouse,a trackball, a trackpad, or cursor direction keys can be coupled to thebus for communicating direction information and command selections tothe processor and to control cursor movement on the display 921.

Camera and microphone arrays 923 are coupled to the bus to observegestures, record audio and video and to receive visual and audiocommands as mentioned above.

Communications interfaces 925 are also coupled to the bus 901. Thecommunication interfaces may include a modem, a network interface card,or other well known interface devices, such as those used for couplingto Ethernet, token ring, or other types of physical wired or wirelessattachments for purposes of providing a communication link to support alocal or wide area network (LAN or WAN), for example. In this manner,the computer system may also be coupled to a number of peripheraldevices, other clients. or control surfaces or consoles, or servers viaa conventional network infrastructure, including an Intranet or theInternet, for example.

It is to be appreciated that a lesser or more equipped system than theexample described above may be preferred for certain implementations.Therefore, the configuration of the exemplary systems 800 and 900 willvary from implementation to implementation depending upon numerousfactors, such as price constraints, performance requirements,technological improvements, or other circumstances.

Embodiments may be implemented as any or a combination of: one or moremicrochips or integrated circuits interconnected using a parentboard,hardwired logic, software stored by a memory device and executed by amicroprocessor, firmware, an application specific integrated circuit(ASIC), and/or a field programmable gate array (FPGA). The term “logic”may include, by way of example, software or hardware and/or combinationsof software and hardware.

Embodiments may be provided, for example, as a computer program productwhich may include one or more machine-readable media having storedthereon machine-executable instructions that, when executed by one ormore machines such as a computer, network of computers, or otherelectronic devices, may result in the one or more machines carrying outoperations in accordance with embodiments of the present invention. Amachine-readable medium may include, but is not limited to, floppydiskettes, optical disks, CD-ROMs (Compact Disc-Read Only Memories), andmagneto-optical disks, ROMs (Read Only Memories), RAMs (Random AccessMemories), EPROMs (Erasable Programmable Read Only Memories), EEPROMs(Electrically Erasable Programmable Read Only Memories), magnetic oroptical cards, flash memory, or other type of media/machine-readablemedium suitable for storing machine-executable instructions.

Moreover, embodiments may be downloaded as a computer program product,wherein the program may be transferred from a remote computer (e.g., aserver) to a requesting computer (e.g., a client) by way of one or moredata signals embodied in and/or modulated by a carrier wave or otherpropagation medium via a communication link (e.g., a modem and/ornetwork connection). Accordingly, as used herein, a machine-readablemedium may, but is not required to, comprise such a carrier wave.

References to “one embodiment”, “an embodiment”, “example embodiment”,“various embodiments”, etc., indicate that the embodiment(s) of theinvention so described may include particular features, structures, orcharacteristics, but not every embodiment necessarily includes theparticular features, structures, or characteristics. Further, someembodiments may have some, all, or none of the features described forother embodiments.

In the following description and claims, the term “coupled” along withits derivatives, may be used. “Coupled” is used to indicate that two ormore elements co-operate or interact with each other, but they may ormay not have intervening physical or electrical components between them.

As used in the claims, unless otherwise specified the use of the ordinaladjectives “first”, “second”, “third”, etc., to describe a commonelement, merely indicate that different instances of like elements arebeing referred to, and are not intended to imply that the elements sodescribed must be in a given sequence, either temporally, spatially, inranking, or in any other manner.

The drawings and the forgoing description give examples of embodiments.Those skilled in the art will appreciate that one or more of thedescribed elements may well be combined into a single functionalelement. Alternatively, certain elements may be split into multiplefunctional elements. Elements from one embodiment may be added toanother embodiment. For example, orders of processes described hereinmay be changed and are not limited to the manner described herein.Moreover, the actions any flow diagram need not be implemented in theorder shown; nor do all of the acts necessarily need to be performed.Also, those acts that are not dependent on other acts may be performedin parallel with the other acts. The scope of embodiments is by no meanslimited by these specific examples. Numerous variations, whetherexplicitly given in the specification or not, such as differences instructure, dimension, and use of material, are possible. The scope ofembodiments is at least as broad as given by the following claims.

What is claimed is:
 1. An apparatus comprising: at least one memory;machine readable instructions; and processor circuitry to execute themachine readable instructions to: identify a gesture based on one ormore outputs of one or more sensors, the gesture to indicate aninteraction with content on a display screen, the content associatedwith an application; determine a view of the content on the displayscreen as being one of a first content view or a second content viewbased on an orientation of the content on the display screen, thecontent having a first orientation in the first content view, thecontent having a second orientation in the second content view, thefirst orientation different than the second orientation; in response tothe view of the content being the first content view, associate thegesture with a first command; in response to the view of the contentbeing the second content view, associate with gesture with a secondcommand; and cause the application to output one of (a) a first responsebased on the first command or (b) a second response based on the secondcommand, the first response different than the second response.
 2. Theapparatus of claim 1, wherein the first response includes a firstinteraction with the content and the second response includes a secondinteraction with the content, the first interaction different than thesecond interaction.
 3. The apparatus of claim 1, wherein the firstcommand is to cause the application to modify the content to generatefirst modified content and the second command is to cause theapplication to modify the content to generate second modified content,the first modified content different than the second modified content.4. The apparatus of claim 1, wherein the view of the content isinitially the first content view and the processor circuitry is to:detect a change from the first content view to the second content view;and cause the application to output the second response based on thedetection of the change.
 5. The apparatus of claim 1, wherein the one ormore outputs of the one or more sensors includes image data.
 6. Theapparatus of claim 5, wherein the processor circuitry is to identify thegesture based on a comparison of the image data to reference gesturedata.
 7. An apparatus comprising: means for displaying content; meansfor detecting a gesture of a user during presentation of the content onthe displaying means; and means for processing to: identify a view ofthe content on the displaying means as a first content view or a secondcontent view based on an orientation of the content on the displayingmeans, the content having a first orientation in the first content view,the content having a second orientation in the second content view, thefirst orientation different than the second orientation; associate thegesture with a first command or a second command based on the view ofthe content; and cause the displaying means to display (a) a firstinteraction with the content based on the first command or (b) a secondinteraction with the content based on the second command, the firstinteraction different than the second interaction.
 8. The apparatus ofclaim 7, wherein the detecting means includes a camera.
 9. The apparatusof claim 7, wherein the detecting means includes a touch sensor carriedby the displaying means.
 10. The apparatus of claim 7, wherein the firstcontent view is associated with a first window on the displaying means,the second content view is associated with a second window on thedisplaying means, further including means for tracking an eye of theuser, the processing means to: determine a gaze direction of the userrelative to the first window or the second window based on one or moreoutputs from the eye tracking means; and cause the displaying means todisplay the first interaction or the second interaction based on thegaze direction.
 11. The apparatus of claim 7, wherein the view of thecontent is initially the first content view and the processing means isto: detect a change from the first content view to the second contentview; and cause the displaying means to display the second interactionin response to the detection of the change.
 12. At least one of anoptical disk, an optical card, a magnetic-optical disk, random accessmemory (RAM), read-only memory (ROM), Flash Memory, or magnetic memorycomprising instructions that, when executed, cause at least oneprocessor to at least: identify a gesture based on one or more outputsof one or more sensors, the gesture to cause a first interaction or asecond interaction with content on a display screen; associate thegesture with the first interaction or the second interaction based on anorientation of the content on the display screen; and cause the displayscreen to present modified content representative of (a) the firstinteraction with the content or (b) the second interaction with thecontent, the first interaction different than the second interaction.13. The at least one of an optical disk, an optical card, amagnetic-optical disk, random access memory (RAM), read-only memory(ROM), Flash Memory, or magnetic memory of claim 12, wherein theorientation is initially a first orientation and the instructions causethe at least one processor to: detect a change from the firstorientation to a second orientation; and cause the display screen topresent second modified content in response to the detected change. 14.The at least one of an optical disk, an optical card, a magnetic-opticaldisk, random access memory (RAM), read-only memory (ROM), Flash Memory,or magnetic memory of claim 12, wherein the instructions cause the atleast one processor to identify the gesture based on a comparison ofimage data to reference gesture data.
 15. An apparatus comprising: atleast one memory; machine readable instructions; and processor circuitryto execute the machine readable instructions to: identify a gesturebased on one or more outputs of one or more sensors, the gesture tocause a first interaction or a second interaction with content on adisplay screen; associate the gesture with the first interaction or thesecond interaction based on an orientation of the content on the displayscreen; and cause the display screen to present one of (a) the firstinteraction or (b) the second interaction with the content, the firstinteraction different than the second interaction.
 16. The apparatus ofclaim 15, wherein the orientation is initially a first orientation andthe processor circuitry is to: detect a change from the firstorientation to a second orientation; and cause the display screen topresent second modified content in response to the detected change. 17.The apparatus of claim 15, wherein the processor circuitry is toidentify the gesture based on a comparison of image data to referencegesture data.
 18. The apparatus of claim 15, wherein the processorcircuitry is to cause the display screen to present the one of the firstinteraction or the second interaction based on a direction of a gaze ofa user relative to a first window on the display screen or a secondwindow on the display screen.